UFAL/Add CCMM 1.1.0 OAI-PMH crosswalk (ccmm-xml metadataPrefix) - #1262
UFAL/Add CCMM 1.1.0 OAI-PMH crosswalk (ccmm-xml metadataPrefix)#1262milanmajchrak wants to merge 9 commits into
Conversation
There was a problem hiding this comment.
Pull request overview
This PR implements a new OAI-PMH crosswalk for exporting DSpace metadata in CCMM (Czech Common Metadata Model) 1.1.0 format, required for NMD/NRP (National Repository of Gray Literature) compliance in the Czech Republic. The implementation follows the standard DSpace OAI-PMH crosswalk pattern used by other metadata formats in the repository.
Changes:
- New XSL transformation stylesheet mapping DSpace internal metadata to CCMM 1.1.0 XML schema
- Configuration updates to register the new crosswalk in both Default Context and openaire_data Context
- Comprehensive test suite with 14 unit tests and dedicated test data file
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 14 comments.
| File | Description |
|---|---|
| dspace/config/crosswalks/oai/xoai.xml | Registers ccmm format (metadataPrefix: ccmm-xml) in Default and openaire_data contexts |
| dspace/config/crosswalks/oai/metadataFormats/ccmm.xsl | 633-line XSLT 2.0 stylesheet implementing DSpace to CCMM metadata transformation with fallback handling for required fields |
| dspace-oai/src/test/java/org/dspace/xoai/tests/stylesheets/CcmmXslTest.java | Test class with 14 tests validating core CCMM elements (identifiers, creators, subjects, license, etc.) |
| dspace-oai/src/test/resources/xoai-ccmm-test.xml | Test fixture simulating typical NMD/NRP dataset metadata with Czech academic repository data |
- Extract hardcoded fallback strings as XSLT variables at top of stylesheet (Unknown Repository, http://unknown.repository, Untitled, unspecified, 9999) - Add detailed comment explaining Handle URL extraction assumptions - Add comment explaining publisher to Distributor role mapping rationale - Fix time_reference fallback: use dc.date.accessioned instead of hardcoded 9999-01-01 when issued/available dates are missing - Improve FormatDate fallback: validate first 4 chars are digits before using as year, output empty ccmm:date element for invalid input - Improve identifier fallback logic: check if Handle-pattern URIs were actually found rather than just checking for raw element presence - Add 6 new fallback scenario tests (title, publication_year, subject, identifier, repository name, license) with dedicated minimal test fixture - All 21 tests pass (20 CCMM + 1 QDC)
Verified against all 2628 live LINDAT records; every output validates against CCMM 1.1.0 in XSD 1.1 (0 invalid). - fix XPTY0004: doc:field[@name='value'] returns a sequence, and records with two language variants of a field passed it to concat()/contains(). 425 of 2628 records aborted; the resulting OAIException becomes HTTP 500 for the whole ListRecords page, so all 32 pages failed and nothing was harvestable - replace the model.ccmm.cz IRIs, which all return 404, with the registers CCMM 1.1.0 names in its own specification (en/dsv.ttl scope notes): COAR resource types and access rights, vocabs.ccmm.cz AgentRole / TimeReference / DescriptionType, EU Publications Office for languages - dc.publisher now maps to AgentRole/Publisher; Distributor does not exist at that level of the codelist - derive access_rights from others/access-status, which DSpace computes in DefaultAccessStatusHelper from the actual READ policies. The previous branches were dead (local.embargo.termslift occurs 0 times) or inverted (others/@restrictedaccess is set on 24 records, all of them open access). Correct for 746 of 2628 records before, 2628 of 2628 now - handle the CLARIN "0000" unknown-date convention and local.approximateDate.issued the way ClarinDateService does in the UI: the local field replaces dc.date.issued, a range becomes a time_interval and the original string is kept verbatim in ccmm:date_information - emit exactly one Issued time reference plus a Created one, as the CCMM profile requires; publication_year and the Issued year now always agree - stop emitting an invented licence IRI; fall back to others/cc and otherwise leave ccmm:license empty and move the wording to terms_of_use/description - the minimal test fixture used to produce schema-invalid output (no time_reference at all); it no longer can Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 6 out of 6 changed files in this pull request and generated 9 comments.
Suppressed comments (3)
dspace/config/crosswalks/oai/metadataFormats/ccmm.xsl:174
original_repositoryis populated from the item identifier, so the commonhdl.handle.net/...form makes every record identify its item resolver as the repository. XOAI already supplies the actual repository URL atrepository/url(ItemUtils.java:347); use that value instead.
<xsl:when test="doc:metadata/doc:element[@name='dc']/doc:element[@name='identifier']/doc:element[@name='uri']/doc:element/doc:field[@name='value']">
<xsl:variable name="uri" select="(doc:metadata/doc:element[@name='dc']/doc:element[@name='identifier']/doc:element[@name='uri']/doc:element/doc:field[@name='value'])[1]"/>
dspace-oai/src/test/java/org/dspace/xoai/tests/stylesheets/CcmmXslTest.java:125
- This assertion exceeds the enforced 120-character line limit (
checkstyle.xml:47-52). Wrap the expected organization name.
equalTo("Charles University, Faculty of Mathematics and Physics, Institute of Formal and Applied Linguistics"))));
dspace-oai/src/test/java/org/dspace/xoai/tests/stylesheets/CcmmXslTest.java:182
- This XPath exceeds the repository's enforced 120-character Java line limit (
checkstyle.xml:47-52). Split it into concatenated segments.
"//ccmm:dataset/ccmm:metadata_identification/ccmm:qualified_relation/ccmm:relation/ccmm:organization/ccmm:name",
- original_repository/iri named the item itself on every one of the 2628 records: it was derived with substring-before(uri, '/handle/'), which does not match LINDAT's hdl.handle.net URIs, so the otherwise branch emitted the whole item URI. XOAI already carries repository/@url; use it. - xml:lang was hardcoded "en" on subjects, alternate titles, descriptions and the rights wording. XOAI nests every value inside an element named after its language qualifier, so take the tag from there and fall back to "en" only for the unqualified "none" wrapper. Both covered by new tests. Re-checked over all 2628 live LINDAT records: still 2628/2628 valid against CCMM 1.1.0 in XSD 1.1. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ields it dropped
An adversarial audit over all 2628 live LINDAT records found 30 defects. They fall into three
groups, and the first is the one that matters: the feed was stating things the source does not
support.
False statements, now fixed:
- access_rights was taken from others/access-status alone. DefaultAccessStatusHelper derives that
from the primary bitstream's READ policies, which on a CLARIN repository does not say whether
the resource can be downloaded: the gate is the licence category in dc.rights.label plus
others/restrictedAccess. 1216 of 2628 records carried a wrong statement - 1161 anonymously
downloadable datasets published as "restricted access", 55 login-gated ones as "open access".
Verified against the live repository: 12/12 sampled "restricted" records return HTTP 206 to an
anonymous ranged GET, 11/12 "open" ACA/RES records return 401.
- Every date_type label was the register's concept id ("Issued") rather than its prefLabel
("Date Issued") - 10512 labels, none correct.
- conforms_to_standard identified the profile by this stylesheet's own XML namespace under the
label "CCMM 1.1", a string that appears nowhere in the CCMM release.
- license/label carried the CLARIN access category (PUB/ACA/RES) instead of the licence name,
which was demoted to a description. 1932 records.
- Hierarchical vocabulary paths ("People::Masaryk ...") were published verbatim as subject titles;
the leaf is now the title and the path a classification_code.
- Agent type was hard-coded per source field, so film studios and ANVL placeholders were published
as people. Names are now typed from the value, and placeholders produce no agent at all.
- publication_year took min() over the approximate years even when the record stated an exact
issue date; a comma enumeration ("1920, 1932") became a continuous interval; a "0000" issue date
fell through to the ingest timestamp; Created ignored the creation year the record states.
- RelationType/Replaces and IsReplacedBy do not exist in the register - the concepts are
Obsoletes and IsObsoletedBy. Every emitted IRI and label was rechecked against the register.
Fields that were being dropped, now read: the contact person and the repository contact, the
owning collection, dc.identifier.other, dc.source.uri, local.demo.uri, every dc.relation
qualifier, dc.format, dc.coverage.placeName, given/family name parts, language names,
local.sponsor, local.additional.metadata, local.size.info, local.refbox.format and the metashare
tree. Bitstreams of the ORIGINAL bundle now become distributions with format, size and checksum.
Portability, none of which changed LINDAT output: ISO 639-1 and region-tagged language codes are
accepted, the stock DSpace type list is mapped, dates are proved castable before being emitted,
the funding guard names the whole grantAgreement prefix, and a repository with no url is no longer
described as living at each item's own address.
Tests: 70 methods over 9 fixtures, including a stock-DSpace record, every date shape DSpace
stores, and the two CLARIN access shapes. Element order is asserted directly - the CCMM schema is
an xs:sequence and validating against it would pull xml.xsd and GML over the network at build
time - plus closed-set guards so no off-register string can come back. Mutation testing: 45 of 46
single-edit mutants are now caught, against 20 of 72 before.
Re-run over all 2628 live records: 2628/2628 valid against CCMM 1.1.0 in XSD 1.1.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
XmlMatcherBuilder builds its NamespaceContext from a plain map with no built-in prefixes, so JAXP cannot resolve the reserved "xml" prefix and ccmmXmlLangFollowsTheXoaiLanguageWrapper failed with "Prefix must resolve to a namespace: xml". Register it alongside ccmm. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…tested Copilot asked for coverage of the embargo/restricted access paths and of the FormatDate variations. Most of what it listed is already asserted, and one of its premises was wrong - the crosswalk never reads local.embargo.termslift, embargo comes from others/access-status. Three branches genuinely had no assertion, so they get one: - others/restrictedAccess=true on a PUB record with files must still be restricted access; the download gate outranks the licence category. 19 live records take that path. - others/access-status=embargo must reach the COAR embargo term, not the restricted fallback. - a dc.date value carrying a time must stay a ccmm:date_time; time_instant is a choice, and truncating it into ccmm:date would silently drop the time. 73 test methods over 11 fixtures. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
license-maven-plugin checks src/** and fails the dspace-oai build without it. The check runs in the verify phase, so it only surfaced once the unit tests stopped failing first. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every record already validated against dataset/schema.xsd, which is why none of this showed up
as invalid. The CCMM specification states requirements the XSD has no way to express - they live
in en/dsv.ttl as scope notes and cardinality - and a conformance check reads those, not the XSD.
All 89 scope notes and 92 cardinality statements were extracted and checked against the output of
all 2628 live LINDAT records.
The one the specification calls mandatory:
- FRASCATI FORD. Dataset.hasSubject: "At least one subject must be a value from FRASCATI FORD
vocabulary. For that subject use reference codelist .../SubjectCategory/ as the Subject scheme."
No record carried one. The field is read off dc.type: a tool or service is computer science
(10201), a newsreel clip is History (60101) - on this corpus every dc.type "clip" is a segment
of a digitised 1943 newsreel held as a historical source, not a language resource - and
everything else is Linguistics (60203), which is what the repository collects. Codes and both
prefLabels come from the live register, and FORD_DEFAULT_CODE is the single value another
deployment changes. 1161 History, 1136 Linguistics, 331 Computer sciences.
Statements the output was making that the source does not support:
- access_url pointed at the bitstream. accessUrl: "The resource at the access URL contains
information about how to get the Dataset. It shall be the web page (not a document file)", and
the schema's own Czech documentation says to use it "in all cases except a downloadable file
URL". It is now the item's landing page; download_url keeps the file.
- identifier/value carried the resolved URL. CCMM types value as skos:notation - the identifier
WITHIN its scheme - and keeps the resolvable form in iri, the split its own samples use. One
emitter now serves every source, so scheme, notation and IRI cannot disagree between branches.
- The dataset had no iri of its own, so the published record had no resolvable identity.
- license is 1..1 and was serialised as <license/> on 584 records. A void mandatory element tells
a harvester nothing; those records now carry rightsstatements.org UND, which is the standard
statement for rights not stated.
- Language labels were DSpace's own strings, not the register's, and glued words together
("UpperSorbian", "AncientGreek (to 1453)") on 537 elements; their positional pairing could also
slip once duplicate codes were removed. The scope note asks for the register's "label and/or
its 3-letter code" and CCMM's own dataset-mini sample writes primary_language with an iri and
no label, so the IRI alone is published.
- media_type published IANA IRIs for six types RFC 6838 reserves as unregistered (application/
x-gzip and friends); those IRIs 404. They are suppressed, and the mandatory EU format element
carries the file type - which is the scope note's own "otherwise Format MAY be used" branch.
- format carried the IANA media type where Format.label asks for a label from the EU file-type
codelist. format is now the EU concept, media_type stays IANA. All 9125 distributions map to a
real EU concept; every code was verified by content, because that register answers 200 for any
path.
Values that were not what they claimed to be:
- 203 byte-identical sibling elements across 161 records, because DSpace stores one value under
several language wrappers and two sources can carry the same value. Deduplication now happens
where each block is assembled, so it also catches the cross-source repeats.
- "Germany||United States" was published as one place name; it is two.
- 20 alternate titles said nothing about how they differ from the title.
- dc.relation.uri was listed as a qualifier but had no term, so those related resources came out
untyped; the register has Other for exactly that.
- An iri only had to start with a URL, so "http://a/ Book and enhanced publication" was published
as an identifier. The whole value now has to be one URI.
- A packed contact field put an affiliation in the email slot.
- A second dc.title never became an alternate title. The predicate was
doc:field[@name='value'][position() > 1], but position() counts within each parent and every
XOAI language wrapper holds exactly one field, so it was always 1. Two titles only worked when
both sat in the SAME wrapper, which XOAI does not do for values differing in language. Latent
on this corpus - no record has two dc.title - but wrong, and the new fixture exercises it.
Measured after the change over all 2628 records: 2628 valid, every counter zero, 95 tests and
187 assertions green, 67 of 68 stylesheet mutants killed, 23 portability inputs clean. The
surviving mutant edits a FormatDate branch its callers make unreachable by pre-filtering.
Not changed, and why: eight records carry U+FFFD in title and description text. That is in the
source data - eight records in, eight records out - so guessing the intended character would be
inventing content. It belongs in the repository's metadata.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
ccmm-transformacie.html
|
Problem description
Add CCMM 1.1.0 OAI-PMH crosswalk — exports metadata as CCMM 1.1.0 over OAI-PMH (
metadataPrefix: ccmm-xml), required for NMD/NRP.New
ccmm.xsl, registered inxoai.xml(Default + openaire_data context), plus unit tests and test data.Analysis
Run over all 2 628 live LINDAT records and validated against CCMM 1.1.0 with XSD 1.1:
ListRecordspages a harvester can fetchWhat the feed was getting wrong
Transform error —
doc:field[@name='value']is a sequence; the 425 records carrying two language variants of a field threwXPTY0004.OAIExceptionbecomes HTTP 500 for the whole page, so every one of the 32 pages failed and nothing was harvestable.Access rights were wrong on 1 216 of 2 628 records. They came from
others/access-status, whichDefaultAccessStatusHelperderives from the READ policies of the primary bitstream — on a CLARIN repository that does not say whether the resource can be downloaded. The gate is the licence category indc.rights.label(PUB/ACA/RES) together withothers/restrictedAccess. Verified against the live repository with anonymous rangedGETs: 12/12 sampled records published as "restricted access" return HTTP 206, 11/12 published as "open access" return 401. Those two signals now decide first andaccess-statusis the fallback.Vocabularies — every
model.ccmm.cz/vocabulary/…IRI returned 404. Replaced with the registers CCMM names inen/dsv.ttl. Every emitted IRI and label was then checked against the live register, which answers a real concept with its prefLabel and a bogus one with nothing. That found three more errors: all fourdate_typelabels were the concept id (Issued) rather than the register's prefLabel (Date Issued);conforms_to_standardidentified the profile by this stylesheet's own XML namespace under the label "CCMM 1.1", a string absent from the whole CCMM release; andRelationType/Replaces/IsReplacedBydo not exist — the concepts areObsoletesandIsObsoletedBy.Licence —
license/labelcarried the CLARIN access category (PUB/ACA/RES) on 1 932 records instead of the licence name, which was demoted to a description. A free-textdc.rights.uriwas copied verbatim intolicense/iri.Dates —
publication_yeartookmin()over the approximate years even when the record stated an exact issue date (48 records backdated by up to 39 years); a comma enumeration (1920, 1932) became a continuous interval although the source names two discrete years;dc.date.issued = "0000"fell through to the DSpace ingest timestamp; andCreatedreused the deposit year on 158 records that state a creation year inlocal.additional.metadata.Agents — the type was hard-coded per source field, so film studios, institutions and ANVL placeholders were published as
prov#Person. Names are now typed from the value, packed values are split, placeholders produce no agent at all, andgiven_name/family_nameare emitted for the 15 448 values that carry the split.Subjects — hierarchical vocabulary paths (
People::Masaryk …) were published verbatim as titles. The leaf is now the title and the whole path aclassification_code.Fields that were being dropped
The contact person and the repository contact, the owning collection,
dc.identifier.other,dc.source.uri,local.demo.uri, everydc.relationqualifier,dc.format,dc.coverage.placeName, language names,local.sponsor,local.additional.metadata,local.size.info,local.refbox.formatand the wholemetasharetree. Bitstreams of theORIGINALbundle now becomedistributionentries with format, byte size and checksum.Portability
None of this changes LINDAT output, but the crosswalk ships in DSpace core: ISO 639-1 and region-tagged language codes are accepted (
en,en_US,de-DE), the stock DSpace type list is mapped, every date is proved castable before it is emitted, the funding guard names the wholegrantAgreementprefix instead of anyinfo:URI, and a repository with nourlis no longer described as living at each item's own address.Tests
70 methods over 9 fixtures, including a stock-DSpace record, every date shape DSpace stores, and both CLARIN access shapes. Element order is asserted directly with
preceding-sibling— the CCMM schema is anxs:sequence, and validating against it would pullxml.xsdand GML over the network at build time — plus closed-set guards so no off-register string or unresolvable IRI host can come back. Mutation testing over 46 single-edit mutants: 45 caught, against 20 of 72 before. The one survivor edits aFormatDatebranch that its callers make unreachable.Problems
CCMM asks for at least one subject from the FRASCATI FORD vocabulary. LINDAT has no FORD field, so no record satisfies that yet — it needs a submission-form change, not a crosswalk change.
Not mapped, because none of it describes the resource:
local.branding,local.hidden,local.hasCMDI, thelocal.files.*counters, the deposit licence blob, and bundles other thanORIGINAL. Carried in the source but with nowhere to go in CCMM 1.1.0: the per-file description on a bitstream, sincedistribution_downloadable_filehas no description element.Resolves: ufal#1145
Manual Testing (if applicable)
Copilot review
🤖 Generated with Claude Code